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INTRODUCTION 


In  a  number  of  disciplines,  data  analysts  are  con¬ 
fronted  with  the  problem  of  classifying  an  observation  into 
one  of  the  distinct  groups  when  the  number  of  variables  is 
very  large.  So,  it  is  of  interest  to  find  out  a  smaller  num¬ 
ber  of  important  variables  which  are  adequate  for  discrimina¬ 
tion.  These  variables  maybe  a  subset  of  the  original  vari¬ 
ables  or  certain  linear  combinations  of  the  original  vari¬ 
ables.  The  selection  of  variables  is  important  since  there 
are  situations  where  inclusion  of  unimportant  variables  may 
actually  decrease  the  ability  for  discrimination.  Apart  from 
it ,  it  is  more  feasible  to  analyze  the  data  from  cost  and 
computational  considerations  if  the  number  of  variables  is 
small.  In  this  chapter,  we  discuss  various  procedures  for 
the  selection  of  variables  in  discriminant  analysis. 

In  Section  2,  we  discuss  procedures  to  find  out  whether 
certain  discriminant  coefficients  associated  with  variables 
are  important  for  discrimination  between  two  populations. 

In  Section  3,  generalizations  of  the  above  procedures  for 
several  populations  are  discussed.  ^The  problems  of  testing 
the  hypotheses  on  discriminant  coefficients  using  simultaneous 
test  procedures  will  be  discussed  in  another  paper.  The  pro¬ 
cedures  in  Sections  2  and  3  are  based  upon  using  conditional 

I  s 

distributions.  '-"In  Section  4  we  discuss  various  procedures 
to  determine  the  number  of  important  discriminant  functions.. 
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2.  TESTS  ON  DISCRIMINANT  FUNCTIONS  USING 
CONDITIONAL  DISTRIBUTIONS  FOR  TWO  POPULATIONS 

In  this  section,  we  discuss  procedures  for  testing 
the  hypotheses  on  the  coefficients  of  the  discriminant  func¬ 
tions  associated  with  the  discrimination  between  two  multi¬ 
variate  normal  populations.  Let  the  mean  vector  and  covari¬ 
ance  matrix  of  i  th  multivariate  normal  populations  be  given 
by  yi  and  E.  Now,  consider  the  discriminant  function  a'x  for 
the  two  population  case  where  a'  =  (a^,...,a  )  ■ 
and  x'  *  (x^,.,.,x  ).  If  any  of  the  coefficients  ai  are  zero, 
then  the  corresponding  variables  xi  do  not  make  any  contribu¬ 
tion  for  the  discrimination  between  the  two  populations.  So, 
it  is  of  interest  to  find  out  as  to  which  of  the  coefficients 
are  zero.  Suppose,  we  know  a  priori  that  X1 , . . . ,xq  are  im¬ 
portant  and  we  are  not  sure  of  xq+i>***>xp*  Then,  we  are 
interested  in  testing  the  hypothesis  that  aq+1  =...=  ap  =  0. 

Let  x^  =  (x^ . Xq) ,  Xg  —  (xq+-^ , . . .  ,  xp ) ,  5  —  (6^,...,5p) 

and  let  y^ygjE  be  partitioned  as 


»  ^ 

Hil 

,  «  * 

5i 

,  2  = 

fE  E 

L11  l12 

(Hi2  ■ 

| 

• 

E  E 

l21  L22 

and  a’  =  (a^a^).  Also,  y^.a^.S^  are  of  order  q*l  and  E^ 
is  of  order  q*q.  Let  H  denote  the  hypothesis  that  the 
conditional  mean  vector  of  Xg  given  x1  is  the  same  for  both 
populations.  Also,  let  6  =  Then 

H:  Hl2”®Hll  “  ~22"®u21*  (2.1) 
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Now,  let  A  ■  (Ui-y2) '  E”1  ( Vf2 ^  *  A1  “  ( Hll ~ H 21  ^  J‘ll(yil"H21) 

and  A2>1  *  <  <S 2— 6  5 1  ^  *  ^2«l'^2”®  ^1  ^  *  Tbe  byP0l:hesis  H  is 
equivalent  to  62-0<5j  *0.  It  is  known  (see  Rao  (1946))  that 
H  is  equivalent  to  the  hypothesis  that  aq+1  »...*  a  ■  0  and 
it  is  also  equivalent  to  the  hypothesis  that  A  =  A^ .  This 
can  be  interpreted  as  H  is  equivalent  to  the  hypothesis  that 
the  distance  between  the  two  populations  based  on  the  p  vari¬ 
ables  is  equal  to  the  distance  between  the  populations  on 
the  basis  of  the  first  q  variables. 


Let  “  (xii » *  •  *  »xiq*xi£+l » *  *  *  »xip)  *  (5il’5i2^’ 

(i  *  1,2),  be  distributed  as  multivariate  normal  with  mean 


vector  *  1.2)  and  covariance  matrix  E  where  z il  is 

of  order  qxl.  Then,  the  conditional  distribution  of  zi2 

given  z . .  is  multivariate  normal  with  covariance  matrix 
•*11 

Z2*l  *  Z22  “  Z21Z11Z12  and  fflean  vector  Hi  +  s5i!  .where 


»  ui2  -  8u11.  We  wish  to  test  2i  =  n2*  tbat  is* 

6  -  8  6,  ■  0.  This  can  be  done  by  using  any  of  the  known  procedures. 

•*«  «*•  X 

Let  <ziit ^2^  *  (t  *  1  ,n1),be  t-th  observation  on 

i 

(z|i,z^2).  Then,  the  conditional  model  is  given  by 


Ec(5i2t/5iit)  "  Di  +  s5iit 


(2.2) 


for  i  ■  1 ,2  and  t  *  l,2,...,n^, 
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We  can  test  the  hypothesid  H  by  using  the 


following  statistic: 


A  *  A  A 


•o-l 


Ak  A  A 


F  = 


C(  <5 2-S  )  Se2  (  6 2 )  (n-p-1 ) 


(l+cys'^^Kp-q) 


(2.3) 


,-l 


where  Se2#1  =  Se22  -  se21SellSel2 »  ®  "  Se21Sell’"j  tjl-'~j2 


k  5  ”  Z  .  i  —Z  .  n 


D1 

nj!Jk.  =  S  5jkt  ’  n  =  nx  +  n2,  c  -n-^/n  and 
t) 


Sell  Sel2  ' 

2  ^ 

—  \ 
?ilt  “  5il» 

s  = 

“  1  l 

■Se21  Se22  J 

i=l  t=l 

•  5i2t  “  ?i2*  - 

(5ilt”5il«  ’?i2t~5i2 


The  statistic  F  is  distributed  as  the  central  F  distribution 
with  (p-q, n-p-1)  degrees  of  freedom  when  H  is  true.  The 
hypothesis  H  is  accepted  or  rejected  according  as 


F  <  Fa  (2.5) 

where 

P[F  <  Fa  | H]  =  (1-a  )  .  (2.6) 

The  above  procedure  for  testing  the  hypothesis  aq+1=. . ,=ap=0 
was  proposed  by  Rao  (1946,  1966).  It  is  known  (e.g.,  see 
Kshirsagar  (1972))  that  Rao's  U  statistic  is  related  to  the 
F  statistic  given  in  (2.3). 

The  simultaneous  confidence  intervals  associated  with 
the  above  procedure  are  known  to  be 

|b'(«2"®!l"!2+65l)l  i  ^b'S^.ib  <P"Q>  (l+c^S^^VcU-p-l) 

(2.7) 

for  all  nonnull  b. 


3.  TESTS  ON  DISCRIMINANT  FUNCTIONS  FOR 


SEVERAL  POPULATIONS  USING  CONDITIONAL  DISTRIBUTIONS 

In  this  section,  we  consider  the  problem  of  testing  the 

hypotheses  that  the  discriminant  coefficients  associated  with 

certain  variables  in  the  discriminant  functions  are  zero.  Let 

*1 . xk  be  distributed  independently  as  multivariate  normal 

with  mean  vectors  and  covariance  matrix  E.  Also, 

let  x.  . , ( j=l  ,2  , . . . ,n . ) ,  denote  j-th  independent  observation 
i  J  ^ 

on  x^  Then,  the  between  group  sums  of  squares  and  cross  pro¬ 
ducts  (SP)  matrix  is  given  by 

k 

S  =  l  n . (x.  -x  )(x.  -x  ) ' 

S11  S12 
S21  S22 

where 


ni?i 


Zxi  j  »  ^  ®  n^+ .  .  .  +n^  > 
J 


=  l 


Now,  let  •  .£0  denote  the  eigenvalues  of  the  noncentrality 

matrix  SI  =  AE-1  and  let  \k  =  (v^ ,  .  . .  ,  vip) ,  (i*l,...,p),  de¬ 
note  the  eigenvector  corresponding  to  0.^  where 

k  _  , 

A  «  l  n.(y.-y)(y.j-y)  (3.1) 

i-1  1  ~  ~ 


and  y  *  (n^y  +  .  .  . +nkyk)/n .  Suppose  the  rank  of  SI  is  r.  Then, 

9  «6  =0  and  we  have  r  meaningful  discriminant  func- 

r+i  p 

tions.  The  within  group  SP  matrix  is  given  by 


where  sen  is  of  order  q*q. 
and  Z  as  follows: 


Now,  let  us  partition  y.,  v. 

~  x  '“J 


t  \ 


A11 

> 

M 

to 

_ J 

Z12 

A  = 

,  I  - 

A21 

o 

to 

to 

_ 

Z21 

Z22 

/ 

where  yi;L  and  are  of  order  q*l,  and  and  Z^  are 


of  order  qxq.  Let  Hj  :  Vj2  =  0  for  j  =  l,2,...,r  and 


H  *  n  H,.  It  is  known  (e.g.  see  McKay  (1977)  and  Fujikoshi 

j-1  J 


(1980))  that  the  hypothesis  H  and  the  following  statements  are 
equivalent : 

(3.2 


H 12  "  E21Znyil  “•••"  Hk 2  "  Z21Zliykl 


trAZ-1  =  trAjjSjJ. 


(3.3 


The  hypothesis  given  by  (3.2)  can  be  tested  by  using  various 
known  methods  (e.g.,  see  Kshirsagar  (1972)  and  Rao  (1973)) 
like  Roy’s  largest  root  test,  likelihood  ratio  test,  etc. 


4.  TESTS  FOR  THE  NUMBER  OF  IMPORTANT 
DISCRIMINANT  FUNCTIONS 

It  is  well  known  that  Fisher's  linear  discriminant  func¬ 
tions  are  the  best  for  discrimination  among  all  linear  func¬ 
tions  of  the  original  variables.  In  this  section,  we  review 
some  procedures  for  the  selection  of  important  discriminant 
functions . 

We  know  that 

('tl~-~P-~1-)  E(S  S"1)  =  I  +  — - —  =  Z*  (4.1) 

(k-1)  e  (k-1 ) 

Let  be  the  eigenvalues  of  I*.  Then  X^  =  1  +  (0i/(k-l)) 

where  6^^  was  defined  in  Section  3.  The  p  discriminant  func¬ 
tions  are  v.!x . v'x  where  x  is  a  vector  of  observations 

~  I  -p- 

on  the  p  variables,  v^x  is  i~th  most  important  discriminant 
function  and  was  defined  in  Section  3.  The  problem  of 

testing  for  the  rank  of  the  noncentrality  matrix  ft  is  equiva¬ 
lent  to  the  problem  of  testing  for  the  number  of  important 
discriminant  functions.  It  is  also  equivalent  to  testing  the 
following  hypothesis  on  certain  structural  relations  among  the 
components  of  the  mean  vectors: 

AUf  3  C ,  (i  »  1 ,2  . ,k)  (4.2) 

where  A:  sxp  and  £:  pxl  are  unknown  and  the  rank  of  A  is  s. 

The  above  hypothesis  implies  that  the  points  lie 

in  a  r-dimensional  space  where  r  ■  p  -  s. 
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Now,  let  denote  the  hypothesis  that  X^  =  1. 

Then  the  hypothesis  (4.2)  is  equivalent  to  the  hypothesis  that 
P 

n  is  true.  The  problem  of  testing  for  the  rank  of  P  w„s 

i=r+l 

originally  considered  by  Fisher  (1938).  Fisher’s  test  to 
test  the  hypothesis  that  the  rank  of  P  is  r  is  based  upon 
the  statistic 

T1  *  (W---+V  <4 

where  ^r+i  >  •  •  •  >  are  t*ie  eigenvalues  of  SS~^  . 

We  can  test  for  the  rank  of  P  within  the  framework  of 
simultaneous  test  procedures  as  follows.  Accept  or  reject 


Hi ( i=r+l , . . . ,p)  according 


where 


i  -  i  C 

i  >  a 


'iVii'JVi1  -  d-°). 


If  Ht  is  accepted  but  Ht_1  is  rejected,  then  the  rank  of 
2  is  (t-1).  But,  the  distribution  of  ^r+1  involves  X^,...,Xr 
as  nuisance  parameters  even  when  Hr+1  is  true  and  so  the  exact 
values  of  c  cannot  be  computed.  If  we  know  in  advance  that 
Ht,...,Hp,  (t  >  r+1),  are  true  then  we  test  Hr+i > • • • >Ht_i 
only.  In  many  situations,  it  is  of  interest  to  test  , . . . ,H 
simultaneously  since  we  don't  know  in  advance  that  Hr  is  not 
true.  In  these  situations,  we  accept  or  reject  HA  according 


li  S  ca  1 


where 


P[*l  1  c  illL]  -  (1-a) 


Here,  we  note  that  the  hypotheses  H^,...,H  are  nested.  For 
example,  implies  Hi+1,...,Hp.  When  is  true,  the  exact 
distribution  of  2.^  was  given  in  Krishnaiah  and  Chang  (1971). 

For  percentage  points  of  the  distribution  of  in  the  null 
case,  the  reader  is  referred  to  Krishnaiah  (1980)  and  Pillai 
(1960).  A  review  of  the  literature  on  the  distributions 
of  individual  roots  and  certain  functions  of  the  eigenvalues 
is  given  in  Krishnaiah  (1978)  and  Muirhead  (1978). 

Fang  and  Krishnaiah  (1982)  derived  asymptotic  nonnull 
distributions  of  certain  functions  of  the  eigenvalues  of 
some  random  matrices  when  the  underlying  distribution  is  not 
normal . 

The  likelihood  ratio  statistic  for  testing  the  hypothesis 
Hr+1  is  known  to  be 

P  -n/2 

L..  =  n  (1+H.)  D  .  (4.8) 

1  i=r+l  1 

For  large  samples,  -21ogL1  is  distributed  approximately  as 

n(i  )  ■  nT. .  It  is  known  (see  Hsu  (1941a))  that 

r+i  p  i 

nT.^  is  distributed  asymptotically  as  chi-square  with  (p-r)(k-l-r) 

degrees  of  freedom  when  n, 's  tend  to  infinity.  Bartlett  (1947) 

P  1 

showed  that  T0  =  C.  T  log(l+2..)  is  distributed  as  chi- 
*  i-r+1  1 

square  with  (p-r)(k-l-r)  degrees  of  freedom  where  the  correc¬ 
tion  factor  C1  is  given  by  =  (n-l-(p+k) /2) .  The  chi-square 
approximation  to  the  distribution  of  the  statistic  T2  is  better 
that  the  corresponding  approximation  to  nT^.  Lawley  (1959) 
suggested  a  modified  correction  factor  but  it  involves  nuisance 
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I 


! 

{ 


J 


i 

1 

I 

I 


C 


parameters.  For  further  details  on  the  likelihood  ratio  test 
for  the  rank  of  .1,  the  reader  is  referred  to  Kshirsagar  (1972) 
and  Rao  (1973).  Anderson  (1951b)  derived  the  likelihood  ratio 
test  statistic  for  testing  the  rank  of  regression  matrix  under 
the  multivariate  regression  model. 

In  general  ,  we  test  the  hypothesis  by  using 

u(l  i  ),  a  suitable  function  of  l  >  as  follows 

r+1  p  r+1  p 

Accept  or  reject  according  as 


v(ir+l’  •  •  •  $  Cci2 


where 


P^(2r+1 . ^p)  i  Ca2  I  Hr+1  ^  “  (1~a) 


(4.9) 


(4.10) 


Some  special  cases  of  tK  J,r+1 ,  .  .  .  ,  2p )  are  .  ( 2-r+i+ •  •  •  )  . 

etc . 

We  now  review  some  known  results  on  asymptotic  distribu¬ 
tions  of  2,^'s  and  certain  functions  of  these  eigenvalues. 

Let  2,^> .  .  .  >lv  ,(r  £  v  <  p)  ,  be  the  nonzero  eigenvalues  of 
SS~^ .  Also,  let  the  eigenvalues  of  n  have  multiplicities  as 
below. 


e 

9 


1 

Pj  +  1 


(4.11) 


o 

'P 


* 

t-1 


+  1 


e 


p*  +i 


n<5 


t 


0 


10 


* 


where  p*  =  P1+...+pj,  ( j-1,2 , . . . ,t+l) ,  r=p*  ,  v  =  p*+1 ,  and 

p*  =  0.  In  addition,  let 
o 


u  =  /n  (2«* +«„)-*(!  -«„) 
n  n 


Ur+j  =  n£r+j 


(4.12) 


where  h  =  l,2,...,t,  3^=  pj^+l  »  •  •  •  »  p£  and  j-l,...,v-r. 
Also,  let  ni=  nQq  i  for  i  =  l,2,...,k.  Then,  the 
limiting  distribution  of  u,  ,  .  .  .  ,u  ,  as  n  -*-<»,  derived  by  Hsu 

J.  V  o 

(1941b)  is  given  by 


t+1 

((up  . .  •  ,uv)  =  n  n^(uvi* 
j=i  ‘ 


pj-i+i 


■V 


(4.13) 


where  r\ .(*)/ j  =  l,2,...,t),  denotes  the  joint  density 

J 

of  the  eigenvalues  of  A.  and  the  elements  of  A . :  p*  *  p*  are 

J  J  J  j 

distributed  independently  as  normal  with  mean  zero. The  variances  of  th€ 
diagonal  elements  of  A.  are  equal  to  one  whereas  the  variances  of  the 
off-diagonal  elements  are  equal  to  £.  Also,  q  ^(#) 
is  the  joint  density  of  the  eigenvalues  of  A^+^ i . (v-r)x(v-r) 
where  is  distributed  as  the  central  Wishart  matrix  with 

(k-l-r)  degrees  of  freedom  and  E(At+1)  =  (k-r-l)Iv_r.  Here 
Al’',,,At+l  are  distributed  independent  of  each  other.  Expres¬ 
sions  for  the  densities  of  the  eigenvalues  of  A.  ( j  = 1 ,2  ,  .  .  .  ,t) , 

J 

and  At+1  were  given  in  Hsu  (1941b).  The  asymptotic  joint  den¬ 
sity  of  u^,...,u  given  by  (4.13)  was  derived  by  Anderson 
(1951a)  by  a  different  method. 
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Now  let  @i  =  mB^,  (i  =  l,2,...,r),  0 r+1=  •  •  •  =  6p=0  where  m  is 

a  suitable  correction  factor  and  B^'s  are  constants.  Then, 

Fujikoshi  (1976)  derived  approximations  to  the  distributions 

_2 

of  miTi » (1  “  1 >2 »3)  up  to  terms  of  order  m^  where 

T,  =  ?  log(  1  +  2, . )  (4.14) 

1  j-r+1  J 


T 


2 


? 

j=r+l 


(4.15) 


T 


3 


j 


<V(1+V> 


(4.16) 


and  m^nu,  and  m3  are  suitable  correction  factors.  The  first 
terms  in  these  approximations  involve  chi-square  distribution. 

Similar  approximations  can  be  derived  for  various  other  func¬ 
tions  of  ^r+i > • • • » Asymptotic  distributions  of  a  wide- 

class  of  functions  of  in  the  nonnull  cases  were 

1  p 

given  in  Fujikoshi  (1978),  Krishnaiah  and  Lee  (1979)  and  Fang  and  Krishna iah  (1< 
In  some  situations,  we  know  in  advance  that  the  last  few 
eigenvalues  of  5)  are  equal  to  zero.  For  example,  when  p>k-l, 

0k = 0k+1=. . .*0p » 0.  In  these  situations,  it  is  of  interest  to 


test  whether  some  of  the  0^'s(i *  1,2, . . . ,k-l)  are  zero. 


We 


can  test  the  hypotheses  Hj(j  = t ,t+l , . . . ,k-l)  as  follows 


also.  We  accept  or  reject  H  .  ( j  =  t , t+1 , . . . ,k-l)  according 

•J 


as 


Tjk  5  cc 


(4.17) 


where 
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(4.18) 


i 


k-l 

p[Tijk  £  ca;  j  =  t, - k-l  I  _n  Hj)  -  (1-a), 

J  ~ 

T..  =  £./(£.+...+£).  As  pointed  out  earlier,  the  joint  distri- 
JK  J  K  P 

bution  of  , At+i > • • • > »  when  is  true  and  the  sample 

sizes  tend  to  infinity,  is  the  same  as  the  joint  distribution 
of  the  eigenvalues  of  the  central  Wishart  matrix.  Exact  distri¬ 
bution  of  the  ratio  of  the  largest  root  to  the  sum  of  the  roots 
of  the  central  Wishart  matrix  was  considered  in  Schuurmann , 
Krishnaiah  and  Chattopadhyay  (1973)  and  Krishnaiah  and  Schuurmann 
(1974). 

We  now  discuss  the  problems  of  testing  the  hypotheses 
H,  ,...,H  in  an  ad  hoc  sequential  way  using  conditional  distri- 

J, 

butions.  The  hypothesis  is  accepted  or  rejected  according 

as 


£ 


1 


*  cal 


(4.19) 


where 


pC*l  £  c^lHj]  -  (l-a1) 


(4.20) 


If  is  accepted,  we  conclude  that  G  *  0  and  don't  proceed 

further.  If  is  rejected,  we  accept  or  reject  H2  accord¬ 

ing  as 


l2  >  ca2 


(4.21) 


P[£2  £  ca2  1^1  —  cal’®2^  *  ^"a2^‘  (4.22) 

If  Hg  is  accepted,  we  don't  proceed  further.  Otherwise  we 
accept  or  reject  Hg  according  as 


l3  $  ca3 


where 


PCi3  -  ^3^1  -  cal  ’  l2  -  ca2  ;H3]  =  (1-a3) 


(4.23) 


In  general,  if  we  accept  Hi(  we  don't  proceed  further. 
Otherwise,  we  accept  or  reject  Hi+1  according  as 


Ai+1  5  ca , i+1 


where 


—  ca,i+l^j  -  ca  j  ’  J  =  1  >2  » •  •  *  >i  ^-“i+i)*  (4-24) 


Then,  the  overall  type  I  error  to  test  H1 ,H2 , • . . ,Hi+1  se¬ 
quentially  is  given  by  where 


t?0PUt+1  ~  Ca,t+ll£j  -  Caj  ’  J-1*2.  Ht+1]  =  (1-ai+l)  (4.25) 

Chou  and  Muirhead  (1979)  derived  asymptotic  conditional  distri¬ 
bution  of  s-t+2  given  This  distribution  involves 

0 i ,  *  *  *  ,  et  as  nuisance  parameters  even  when  Ht+1  is  true. 

If  we  ignore  the  terms  of  order  0T1,(i  =  l,2,...,t),  then  the 
conditional  distribution  of  fct+1  given  does  not 

involve  nuisance  parameters  when  Ht+^  is  true.  If  we  further 
ignore  the  linkage  factors 

t  P  * 

n  n  (i.  -  4,)* 


i=l  j»t+l 


'i  "j 


the  joint  distribution  of  the  latent  roots  ^+1'***'^)  is  the 
same  as  the  joint  distribution  of  S^S^+Sg)-1  where  and 
S2  are  distributed  independently  as  central  Wishart  matrices 


14 


with  (k-l-t)  and  (n-k-t)  degrees  of  freedom  respectively  and 
E(S^/(k-l-t)  =  E(S2/(n-k-t ))  =  I  t .  In  deriving  the  above 
result ,  Chou  and  Muirhead  assumed  that  the  error  degrees  of 
freedom  is  very  large  but  the  noncentrality  matrix  remain  fixed. 
For  a  discussion  of  the  asymptotic  conditional  distribution 
of  £t+i » •  •  •  >  Ap  given  £^,...,2,^  under  some  other  conditions, 
the  reader  is  referred  to  Muirhead  (1978). 
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